{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Processing Exercise \n",
    "\n",
    "In this exerise, you will learn some building blocks for text processing . You will learn how to normalize, tokenize, stemmeize, and lemmatize tweets from Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fetch Data from the online resource\n",
    "\n",
    "First, we will use the `get_tweets()` function from the `exercise_helper` module to get all the tweets from the following Twitter page https://twitter.com/AIForTrading1. This website corresponds to a Twitter account created especially for this course. This webiste contains 28 tweets, and our goal will be to get all these 28 tweets. The `get_tweets()` function uses the `requests` library and BeautifulSoup to get all the tweets from our website. In a later lesson we will learn how the use the `requests` library and BeautifulSoup to get data from websites. For now, we will just use this function to help us get the tweets we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import exercise_helper\n",
    "\n",
    "all_tweets = exercise_helper.get_tweets()\n",
    "\n",
    "print(all_tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalization\n",
    "Text normalization is the process of transforming text into a single canonical form.\n",
    "\n",
    "There are many normalization techniques, however, in this exercise we focus on two methods. First, we'll converting the text into lowercase and second, remove all the punctuation characters the text.\n",
    "\n",
    "#### TODO: Part 1\n",
    "\n",
    "Convert text to lowercase.\n",
    "\n",
    "Use the Python built-in method `.lower()` for converting each tweet in `all_tweets` into the lower case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 2 \n",
    "\n",
    "Here, we are using `Regular Expression` library to remove punctuation characters. \n",
    "\n",
    "The easiest way to remove specific punctuation characters is with regex, the `re` module. You can sub out specific patterns with a space:\n",
    "\n",
    "```python\n",
    "re.sub(pattern, ' ', text) \n",
    "```\n",
    "\n",
    "This will substitute a space with anywhere the pattern matches in the text. \n",
    "\n",
    "Pattern for punctuation is the following `[^a-zA-Z0-9]`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "counter = 0\n",
    "\n",
    "for tweet in all_tweets:\n",
    "    all_tweets[counter] = re.sub(r'[^a-zA-Z0-9]', ' ', tweet) \n",
    "    counter += 1\n",
    "\n",
    "print(all_tweets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK: Natural Language ToolKit\n",
    "\n",
    "NLTK is a leading platform for building Python programs to work with human language data. It has a suite of tools for classification, tokenization, stemming, tagging, parsing, and semantic reasoning. \n",
    "\n",
    "Let's import NLTK. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os \n",
    "import nltk \n",
    "nltk.data.path.append(os.path.join(os.getcwd(), \"nltk_data\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO: Part 1\n",
    "\n",
    "NLTK has `TweetTokenizer` method that splits tweets into tokens.\n",
    "\n",
    "This make tokenizng tweets much easier and faster. \n",
    "\n",
    "For `TweetTokenizer`, you can pass the following argument `(preserve_case= False)` to make your tokens in lower case. In the cell below tokenize each tweet in `all_tweets` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import TweetTokenizer\n",
    "\n",
    "#  your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Part 2\n",
    "\n",
    "NLTK adds more modularity for tokenization.\n",
    "\n",
    "For example, stop words are words which do not contain important significance to be used in text analysis. They are repetitive words such as \"the\", \"and\", \"if\", etc. Ideally, we want to remove these words from our tokenized lists. \n",
    "\n",
    "NLTK has a list of these words, `nltk.corpus.stopwords`, which you actually need to download through `nltk.download`.\n",
    "\n",
    "Let's print out stopwords in English to see what these words are. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "nltk.download(\"stopwords\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO: \n",
    "\n",
    "print stop words in English"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code is here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO: Part 3 \n",
    "\n",
    "In the cell below use the `.split()` method to split each tweet into a list of words and remove the stop words from all the tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## your code is here \n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming\n",
    "Stemming is the process of reducing words to their word stem, base or root form.\n",
    "\n",
    "### TODO:\n",
    "\n",
    "In the cell below, use  the `PorterStemmer` method from the ntlk library to perform stemming on all the tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "# your code goes here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatizing\n",
    "#### Part 1\n",
    "\n",
    "Lemmatization is the process of grouping together the inflected forms of a word so they can be analyzed as a single item.\n",
    "\n",
    "For reducing the words into their root form, you can use `WordNetLemmatizer()` method. \n",
    "\n",
    "For more information about lemmatzing in NLTK, please take a look at NLTK documentation https://www.nltk.org/api/nltk.stem.html\n",
    "\n",
    "If you like to understand more about Stemming and Lemmatizing, take a look at the following source: \n",
    "https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('wordnet') ### download this part "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO:\n",
    "\n",
    "In the cell below, use the `WordNetLemmatizer()` method to lemmatize all the tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "# your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO: Part 2\n",
    "\n",
    "In the cell below, lemmatize verbs by specifying `pos`. For `WordNetLemmatizer().lemmatize` add `pos` as an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "# your code goes here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Solution\n",
    "\n",
    "[Solution notebook](process_tweets_solution.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
